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Foreword 


Learning outcomes are described as written statements of what a learner is expected 
to know, understand and/or be able to do at the end of a period of learning. At the 
beginning of the 90s, a EU pilot project on ECTS showed that study programmes 
were much easier to compare if they were described in terms of outcomes, instead 
of inputs. Learning outcomes started to gain importance at policy level and have 
consequently been supported by the development of national qualifications frameworks 
(Berlin Communique), the adoption of the ESC, the overarching outcomes-focused 
Qualifications Eramework of the European Higher Education Area (QEEHEA) (Bergen 
Communique) and the European Qualifications Erameworks for Lifelong Learning. 
Today and in the near future student- centered learning and learning outcomes will be 
at the core of ‘implementing Bologna’. 

The importance of learning outcomes will increase for several reasons. Eirstly, 
learning outcomes make qualifications more transparent for students. Then, the range 
of graduates is becoming wider and thanks to learning outcomes, employers may have 
a better understanding of the acquired knowledge, skills and competences in order to 
recruit the most suitable candidate. Learning outcomes benefit for quality assurance 
as they increase transparency and comparability between qualification standards. 
Learning outcomes are also valuable in terms of course design. 

Naturally, objections have also been expressed with regard to student-centred 
learning and learning outcomes. They mainly question the focus on the individual 
learner, the difficulties in the formulation and implementation, and the inappropriate 
approach to higher education and academic study. The results of an ENQA survey on 
quality procedures of quality assurance agencies across Europe and beyond (2008) 
witnessed the education’s shift from a teaching to a learning focus. At the same time 
this survey revealed that learning outcomes have only started to be addressed in 
external quality assurance at programme and institutional level procedures. 

Since then, the methods of agencies take more and more into account learning 
outcomes. 

This report presents four articles based on the ENQA workshop on the theme 
“Quality Assurance and Learning Outcomes” that was held in September 2010 in 
Vienna, Austria. The workshop addressed the question of what stakeholders expect 
from quality assurance agencies in connection with learning outcome orientation. 

The opportunities and challenges of the learning outcome orientation in the higher 
education sector were discussed from different perspectives. The workshop tried to 
define the role that learning outcomes should play in external quality assurance and 
how they can or should be considered within the scope of external quality assurance. 


Achim Hopbach 
President 

European Association for Quality Assurance in Higher Education (ENQA) 


Chapter 1: 

On quality assurance and learning 
outcomes: Evaluating students’ work 
within institutions or institutional 
work with students? 

Lena Adamson, Royal Institute of Technology (KTH), Sweden 


1.1 Introduction 

In September 2009, a new model for quality assurance (QA) was presented to the 
Swedish Government by the National Agency for Higher Education (NAHE). The 
model was based on learning outcomes and will constitute the base for this article, 
although the model as such will not be implemented in Sweden? 

The article is structured around the following topics: a) definitions and levels for 
learning outcomes; b) a brief introduction to the Swedish model; and c) suggested 
models and principles for evaluating intended and achieved learning outcomes. The 
idea is to bring the reader from a fundamental level of these issues all the way down to 
the “knitty gritty” implementation, and to raise some of the basic questions that arise in 
this process. 

The definitions of learning outcomes used throughout the article are as follows: 
Intended Learning Outcomes (ILOs) are written statements of what the student is 
expected to know, understand and be able to do after completion of a learning unit. 
Achieved Learning Outcomes (ALOs) are what individual students have actually 
achieved in relation to the intended learning outcomes of this learning unit. 

BOLOGNA SYSTEM - ILO LEVELS 

EUROPEAN LEVEL - QF-EHEA, EOF 
NATIONAL LEVEL -NQF 

INSTITUTIONAL LEVELS - ILOs PROGRAMME LEVEL 


ILOs course/module level ILOs course/module level ILOs course/module level 

ILOs tasklevel ILOs tasklevel ILOs task level ILOs task level ILOs task level ILOs task level 

Figure 1 


1 The reason was, and still is, a stronger political aim than expressed in the initial assignment to NAHE that the national quality 
assurance system should only evaluate “the results of education” and therefore exclude all evaluation aspects of “educational 
planning and implementation” (p. 12, Focus on knowledge - quality in higher education Government Bill 2009/10:139, adopted 
by the Swedish Riksdag.) 


1.2 Levels of learning outcomes - the Bologna system 

Learning outcomes can be formulated on different levels in the Bologna system, and in 
each discussion, it is important to know which one of these levels you are referring to. 
Further, the Bologna system is a holistic system where all levels need to be integrated. 

The top level is the European level, the Qualifications Framework for the European 
Higher Education Area (QFEHEA). The next level is the national level where some 
countries have level descriptors in their higher education legislation and some do 
not. The following three levels “belong” to institutions and individual teachers; the 
programme level, the course/module level and the individual task level. 

This level structure constitutes one of the cornerstones in the model presented to 
the Government, where level two (the national level) was used as frame of reference, 
against which level four (course/module level) of the programmes under evaluation was 
to be evaluated. 

1.3 The proposed model - background 

The reason for the Swedish Government to assign NAHE to develop a new model for 
quality assurance was the higher education sector’s severe critique of the QA system 
in use. The main critique concerned the lack of transparency and alignment with the 
Bologna process. The Government expressed this in the assignment to NAHE as the 
need for a new QA system to be “more focused on results”. 

The discussion therefore started with defining what constitutes educational results. 

A group of vice chancellors, the chair of the Swedish Student Union and representatives 
from NAHE agreed that the results of higher education (HE), and indeed of any 
education, must concern what students have achieved after the completion of their 
studies. From that, the definition of quality followed easily; quality in Swedish HE 
would be that students achieve the ILOs, as expressed in the qualification descriptors 
(i.e., level two) in the Higher Education Ordinance. In line with that, the basic question 
the new QA model had to answer simply became: '’Do programmes ensure that students 
achieve these learning outcomes?” 

The model was then composed by NAHE, jointly with vice chancellors and their 
quality executives during a series of hearings. Key issues were: transparency and 
predictability, that it should be based on research and experience in both evaluation 
and teaching and learning processes, and finally, on a structural level, supporting the 
regular teaching activities and their planning in faculties. This would make the system 
sustainable and balance the needs for both accountability and improvement (cf. EUA 
Policy Statement, 2010). 

The result was a basic structure of three quality indicators. The first, “Intended 
Learning Outcomes and Assessment”, the second, “Achieved Learning Outcomes” and 
the third, “The Student Experience”^. Each evaluated program would acquire a quality 
profile based on grading from 1 to 3 for each quality indicator. 


2 The full model will not be discussed in this paper, only the parts that refer to ILOs and ALOs. 


Schematic presentation of the new system 

QUALITY INDICATORS 


1. Intended learning outcomes 


a. Aligned teaching and 
progression 

b. Employability 
("usability") 


c. Teacher resources 


2. Achieved learning outcomes 3. Student experiences 

a. Review of independent projects a. Student surveys 


b. Alumni surveys 


Example of a fictive quality profile 



a. Intended learning outcomes a. Achieved learning a. Student experiences 

outcomes 

This character shows an example of an imaginary quality profile. The numbers are credits from the assessment panel of the 
different basis of assessment from a fictive evaluation. 

Figure 2 

1.4 Assessing and evaluating^ learning outcomes - moving closer to reality 

Difficulties in drawing conclusions about educational quality from a learning outcome 
perspective will increase the further away from the teaching and learning situation 
the evaluation is carried out. Between the choice of (quality) audits and evaluations of 
study programmes/subjects there is a need to choose the latter in order to tackle the 
issue at its source, the teacher-student situation where both ILOs and ALOs are applied 
in actual practice. 

1.4.1 ASSESSING OR EVALUATING INTENDED LEARNING OUTCOMES AND 
ALIGNED TEACHING 

Although being the cornerstone in the paradigm shift from teacher-driven to student- 
centred teaching and learning, the use of ILOs are not enough to ensure that these 
are turned into ALOs. In the proposed model, a special focus was therefore laid also 
on what is often labelled aligned teaching/constructive alignment (Biggs, 2003) or the 
learning chain (Gehmlich, 2010). 

Aligned teaching is a way of tying together the different parts of the teaching and 
learning situation for the student in a logical and understandable order. The first step 
is to formulate ILOs and the next to find ways for making these visible, which is the 


3 The term assess will be used here in relation to students/student work whereas evaluate is used in relation to QA. 








design of relevant, fit-for-purpose assessment tasks. After that, decisions on grading 
system and assessment criteria need to be taken. Finally, the teaching and learning 
activities that will enable students to achieve this are designed. 

When teaching is planned in this order, focus is shifted from teacher activities to 
student activities and from the content of the learning unit to the results intended. This 
is a necessary shift to make teaching and learning more student-centred and outcome- 
based. In relation to quality assurance, the same type of logic needs to be applied. 

Thus, the key components for good quality in relation to aligned teaching needs to be 
identified. In the proposed Swedish model, the key components for this part (Indicator 
la) were defined as coverage, assessability, content and level of ILOs and relevancel fitness 
for purpose of examination tasks. 

The first component, coverage, concerns whether the ILOs laid down in the national 
qualification descriptors (or any other benchmark descriptors) are sufficiently covered 
by the combined ILOs of the programme/fields of study under evaluation. 

In order for ILOs to be assessed, they must describe “visible use of knowledge”, or 
competencies, meaning the unique combination of knowledge and skills. The learner 
“knows and understands” but is also “able to do” something with this knowledge. This 
constitutes the heart of the learning outcome paradigm, and therefore, assessability 
(i.e., the use of action verbs) must be seen as the main sign of quality in ILOs. These 
must also have the right content in relation to course objectives and be on the right 
academic level for the programme, not too high or low. 

Third, assessments must be designed in a relevant manner in order to ensure that the 
learner achieves the intended learning outcomes. Thus, the evaluation of assessment/ 
examination tasks with regards to both content and form is necessary. Finally, since the 
existence of written grading criteria has been shown to enhance student learning (eg. 
Rust, Price, & O’Donovan, 2003 ; O’Donovan, Price & Rust, 2008), and is also one of 
the demands in the ESG, this was added as the last area for scrutiny. 

It is important to note that no evaluation of the actual teaching and learning 
activities were included. The reason for this is that, apart from being unnecessarily 
intrusive on institutional autonomy, no overall evidence for any particular teaching 
methods has been provided so far, except for the general active learning strategy. 


DRAFT CHECKLIST EVALUATION OF ILO AND ALIGNED TEACHING 

GRADE 1 

GRADE 2 

GRADES 

1. COVERAGE: 




Do ILO:s cover the descriptors for the exam? 

No 

To a certain extent 

Yes 

2.0UALITY0FIL0S 




Assessable - Do ILO:s describe the visible use of knowledge? 

No 

Reasonably 

Yes 

Do ILO:s clearly address the content of the study module? 

No 

To a certain extent 

Yes 

Do ILO:s have appropriate academic level for the study module? 

No 

Reasonable level 

Yes 

3. OUALITY OF ASSESSMENT RELEVANCE 




Relevant in accordance to content? 




Relevant in accordance to shape and form? 

No 

Reasonable 

Yes 

4. ASSESSMENT CRITERIA: 




Presented in advance? 

No 

- 

Yes 


Figure 3 


1.4.2 ASSESSING OR EVALUATING ACHIEVED LEARNING OUTCOMES 

The initial way of thinking about the methods for including ALOs in a QA system is, 
probably for most of us, to develop a model where subsets of student work are assessed 
in order to generate a general conclusion about the overall quality of the programme 
under evaluation. 

With this strategy, a number of questions immediately arise: a) What is student 
work? ; b) What learning outcomes are connected to them? ; c) Which type of samples 
would be usable as generalisations about the quality of entire study programmes? And 
finally, a more fundamental question: d) Is there a problem that quality evaluators assess 
the same work which has already been assessed by teachers? 

In the Swedish higher education system, the first questions are easy to answer. The 
only student work available on a system level, is the independent degree project every 
student have to do in order to receive her/his exam. All other assignments during the 
studies are legally owned by the students themselves. They are not registered as such 
in any administrative system, and hence, not available on a systematic basis on the 
national level. Regarding the ILOs, guidance for constructing these on both bachelor 
and master level, can be found in the Higher Education Act and the Higher Education 
Ordinance (level two). These would form a basis for more specified learning outcomes 
on programme level to be used in each particular programme evaluation. 

The third question raised is more complicated. In order to draw statistically sound 
generalisations about the quality of individual programmes, random samples are 
necessary, which, in this context, often means very large samples. This constitutes a 
practical problem where the answer lies in the allowed cost of a QA system. In addition, 
it is important to mention that the degree project still represents only a very limited 
portion of the educational programme especially related to programme content. 

The fourth question concerns a role mix-up between the institutions and their 
responsibilities of performing valid, reliable and fair assessment of student work on 
the one hand, while on the other hand, QA organisations base their evaluations on 
assessments of the same student work as a kind of “second opinion”. This constitutes a 
fundamental problem in using ALOs in the QA process. Since role confusion in any system 
often leads to problems, this needs to be resolved. One solution is to twist the question 
to some degree. Instead of evaluating individual student work and then draw general 
conclusions about the quality of the programme, the evaluation can, instead, concern 
whether students with poor results on their degree projects (that is, not achieving the 
intended learning outcomes) are allowed to pass through the system or not. This solves 
a number of issues. Eirst, and most important, although using ALOs, the evaluation 
now concerns the study programme, that is, the quality of work with students instead of 
the quality of student work. This is the very purpose of a QA system. 

Second, the number of degree projects can be reduced, because the evaluation 
can end once an unsatisfactory, or possibly a few unsatisfactory project(s) have been 
observed"^. There is simply no need to go through the remaining sample once these have 
been discovered. The entire sample needs to be examined only in the case where no 
unsatisfactory works are discovered. 

Third, this method does not make statistical generalisations, but instead, base 
the evaluations on empirical observations: either none of the degree projects hold 
suboptimal standards that is, this programme really ensures that students who have 


4 This number depends on how solid the system is. 


passed their exam have actually achieved the ILOs connected to the degree projects. 

Or, one project (or possibly a few) has actually passed the system even though it was not 
up to the required standards, which means that the quality of this programme cannot 
be fully trusted. 

1.5 Conclusions 

Learning outcomes builds on theories and research on active learning which we 
know enhances student learning (e.g. Hake, 1998; Prince, 2004), the main objective 
of all education. This is the natural starting point for quality assurance. It also gives 
evaluators, students and teachers a common language which facilitates the joint project 
of giving every student the optimal conditions for their studies. 

First, in order to include learning outcomes (LOs), we suggest that one necessary 
condition is that the system focuses on programme/subject evaluations instead of 
audits. Further, concerning ILOs, there are few methodological problems, but there is a 
definite need to embed the evaluation of these into a somewhat wider context where we 
suggest aligned teaching. 

In relation to including ALOs in a QA system however, there are both 
methodological and fundamental issues that need to be thought through. First, one 
has to be aware of that evaluating ALOs in a national system will most probably 
always cause workload problems. Further, on the more fundamental level, we have 
suggested that measures are taken in order to construct the system so as to ensure that 
the evaluation concerns not the quality in student work but the quality in teachers' and 
institutions' work with students. This is to ensure that the system evaluates educational 
quality and not student quality. It is well known that the socioeconomic status of 
students explains a large proportion of their results independently of educational 
quality. 

Finally, there still exists a lack of knowledge in higher education about what 
learning outcomes really are and what they can accomplish. This probably explains 
the resistance still found. The challenge is now to reach out to teachers, students 
and evaluators with this knowledge to encourage the use of learning outcomes in a 
competent and creative manner. This can also become a powerful tool in striving to 
further develop the necessary interaction between research, education and innovation 
in the European Higher Education Area. 
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Chapter 2: 

Learning outcomes in external quality 
assurance: The case of Spain 

Maria Becerra, Laureano Gonzalez-Vega & Juan Jose Sabrina^, National Agency for Quality 
Assurance and Accreditation of Spain (ANECA), Spain 


2.1 Introduction 

At this moment, there are 78 universities (of which 50 public and 28 private) with a 
total of 1.600.000 students in Spain. While higher education is regulated by national 
authorities (namely the Ministry of Education), the funding of public universities is 
regionally based, and universities autonomous under the Spanish Constitution. The 
last reforms (Royal Decrees 1393/2007 and 861/2010) remove the old national degrees 
catalogue and give the universities the freedom and flexibility to offer Bachelors and 
Masters instruction in the fields that they internally decide. 

In Spain, accreditation of programmes exists, but not accreditation of institutions 
(though the internal quality assurance systems of institutions can be evaluated). 

Once one university has decided to offer a degree before admitting students, its study 
programme must be first evaluated by the quality assurance agency in charge (ANECA 
or one of the regional quality assurance agencies with such a competence), and then 
authorised by the corresponding region. 

Spanish legislation establishes three different stages in the accreditation process 
(understood as the application of the external quality assurance of Spanish regulations): 
Ex-ante accreditation, follow-up procedure and ex-post accreditation. 

After October 2007 (time of publishing the Royal Decree 1393/2007) and until July 
2010, ANECA has performed the ex-ante accreditation of more than 5000 Bachelor’s 
and Master’s degrees, which constitutes the new offering of Spanish universities for this 
course 2010/2011. 

Ex-ante accreditation is devoted to verify that the design of the considered study 
programme includes the criteria established by law and whose interpretation and 
application was made by ANECA. At this time, ANECA (and other regional agencies) 
are starting the so-called follow-up procedures to check, by non-intmsive means, 
that the institution is implementing the study programme according to the approved 
design. Einally, the ex-post accreditation will certify that the programme has been 
accomplished according to the already approved conditions in the ex-ante accreditation 
process (every six years for Bachelors and every four years for Masters). 

Concerning the ex-ante accreditation procedure, Spanish regulations establishes that 
the design for new degrees have to be evaluated according to a series of quality-based 
criteria to be applied on the basis of the degree proposal presented by the university. 
This proposal contains the following information: 

1. Description of the degree 

2. Justification 

3. Competences 


5 Corresponding author. E-mail: mbecerro@aneca.es, jsobrino@aneca.es and lgonzalez@aneca.es 


4. Student entry and admission 

5. Programme planning 

6. Academic staff 

7. Resources and services 

8. Anticipated outcomes 

9. Quality assurance system 

10. Agenda for implementing the degree 

The rest of this paper is devoted to show where learning outcomes appear in the 
ex-ante accreditation procedure ANECA has applied during the last three years when 
evaluating more than 5000 Bachelor’s and Master’s degrees. 

2.2 Where are learning outcomes integrated in the Spanish accreditation 
process? 

Learning outcomes appear in Spanish regulations through Dublin Descriptors. They 
were included in the Spanish legal framework (Royal Decrees 1393/07 and 861/2010) , 
after which all Spanish degrees need to prove their alignment with the corresponding 
level. The Spanish Qualification Framework for Higher Education (called MECES) has 
been established and defined in terms of learning outcomes (in force from December 
2010). These legal frameworks (both European and Spanish) are integrated as part of 
the evaluation process regarding “competences” (referred to as “learning outcomes” 
here, and when talking about Qualifications Framework). 

Section 3 of the proposal for a concrete degree (Bachelor or Master) must 
explicitly describe the learning outcomes characterising it. And, moreover, the 
competences to be acquired by the students must be consistent with those required 
by the degree and by MECES (or by the Qualifications Framework for the Higher 
Education Area). Competences and learning outcomes must be understood here as 
synonymous but, when talking about modules in a study programme (section 5 of 
the proposal), competences must be described in terms of learning outcomes (which 
implies a different use of both terms depending on the context). The use of the same 
terms with different meanings depending on the context has been a clear source of 
misunderstandings during the entire process. 

For some “professional” degrees (e.g. Medicine or Civil Engineering), “professional” 
learning outcomes must also be taken into account both at the proposal and at the 
evaluation level. The Ministry of Education, in close contact with the corresponding 
professional counterparts at the administration, defines “a priori” these “professional” 
learning outcomes. 

It is worth mentioning here that, for all the competences (or learning outcomes) 
included in section 3, their assessment must be described explicitly. 

Section 5 of the degree proposal is devoted to programme planning. It must display 
a coherent structure of modules and ECTS credits in relation to the intended learning 
outcomes introduced in section 3. Contents, training activities, assessment system and 
prior requirements specified on each module must be appropriate for the module aims. 

Section 8 of the degree proposal must describe the general mechanisms for assessing 
the learning outcomes of students. Section 9 deals with how the internal quality 
assurance system works to monitor the appropriateness of the planned learning 
outcomes assessment. 


Each proposal is evaluated by a commission according to a set of previously defined 
criteria. Those criteria, more closely related with learning outcomes, are the following: 

1. The study programme must constitute a proposal for study that has been designed 
in a coordinated way and takes into consideration the student’s dedication within 
a given period of time. The competences to be acquired by students must be 
assessable and consistent with those corresponding to a Bachelor’s or Master’s 
degree. 

2. The study programme must have a module, course and credit structure that 
is consistent with the defined competences and supported by mechanisms to 
coordinate teaching. The contents, learning activities, evaluation system and 
prerequisites specified in each module must be suitable in order for students to 
acquire the intended competences in the module. 

3. The design must provide a resume with the competences to be acquired and their 
evaluation in the modules that make up the degree design, including the core 
subjects, compulsory and free choice courses, work experience or placement and 
the Bachelor’s or Master’s thesis. 

The commission’s experts do answer a fixed set of questions, including, when 
appropriate, the corresponding comments. Among many other questions, those more 
closely connected with learning outcomes are the following: 

1. Do the defined general competences correspond with those set by Royal Decree 
1393/2007 for awarding of a corresponding Bachelor’s or Master’s degree? 

2. Do the proposed competences match those specified in the literature of national 
and international networks and entities? 

3. Are the competences acquired by the student in the module/course in line with 
the requirements of the degree? 

4. Are the module/course competences specified in terms of learning outcomes? 

5. Do the contents described in the module/course bear a relationship with the 
established competences? 

6. Do the teaching activities in each module/course (considering the teaching/ 
learning methodology) bear a relationship with the competences to be acquired 
by the student? 

7. Do the modules and courses make the study programme consistent with the 
general aims and competences of the degree? 

8. Does the proposed evaluation system allow for assessment to be made of the 
learning outcomes of students? 

What has been described so far shows that learning outcomes play a central and 
prominent role in the evaluation of a degree proposal. 

2.3 What are the results of the Spanish ex-ante accreditation procedure? 

Next steps concerning learning outcomes? 

The results of the ex-ante accreditation procedure appear in the next two tables where 
the results of the evaluation are shown in terms of the five knowledge areas (which are 
defined in the Spanish regulations). These data were gathered in September 2010. 


KNOWLEDGE AREA 

NEGATIVE 

% 

POSITIVE 

% 

TOTAL 

Arts and Humanities 

5 

1,3% 

374 

98,7% 

379 

Sciences 

4 

1,8% 

216 

98,2% 

220 

Health 

16 

4,7% 

321 

95,3% 

337 

Social Sciences and Law 

48 

5,7% 

798 

94,3% 

846 

Engineering and Architecture 

26 

4,0% 

632 

96,0% 

658 

Global 

99 

4,1% 

2341 

95,9% 

2440 


Table 1: Bachelor ex-ante accreditation 


KNOWLEDGE AREA 

NEGATIVE 

% 

POSITIVE 

% 

TOTAL 

Arts and Humanities 

25 

6,2% 

380 

93,8% 

405 

Sciences 

19 

5,6% 

323 

94,4% 

342 

Health 

17 

4,4% 

367 

95,6% 

384 

Social Sciences and Law 

70 

7,3% 

888 

92,7% 

958 

Engineering and Architecture 

24 

4,5% 

508 

95,5% 

532 

Global 

155 

5,9% 

2466 

94,1% 

2621 


Table 2: Master ex-ante accreditation 


The proposals were evaluated at ANECA by several commissions, most members of 
which came from academia, but included also students and professionals. In many 
cases, several iterations were needed in order to obtain a final positive report that 
typically included several recommendations. 

In many cases, a final negative report was based in the fact that the proposed 
learning outcomes did not fit well with the corresponding level. For example, cases 
where the intended learning outcomes were too advanced for a Bachelor or too 
elementary for a Master (according to the level description in terms of learning 
outcomes (LOs) in the Qualifications Framework for the Higher Education Area).^ 

The ex-ante accreditation procedure is continued now by the corresponding 
follow-up. Since society (and not only students) needs to get informed about the 
characteristics of new degrees, this will be, first, based on the need that universities 
commit with the strong need of visibility and transparency. The other two tools of the 
follow-up procedure are the monitoring of how the internal quality assurance systems 
are working, and the comparison between the intended learning outcomes and those 
achieved by the students. 

The follow-up procedure is just starting in Spain since the first (ex-ante) accredited 
degrees started in October 2008, and it is performed by ANECA and the other existing 
regional quality assurance agencies in Spain (under the coordination of the Ministry 
of Education). ANECA is just starting a pilot project to study the difference between 
the intended learning outcomes and the achieved learning outcomes by analysing a 

6 The introduction of new programmes together with the analysis of their learning outcomes has opened an important debate 
within the ANECA evaluation committees that has not been completely solved yet: is it reasonable to offer overly specialised 
training at the Bachelor level? For example, the opportunity for Bachelor proposals like Nanosciences, Bioinformatics, etc. has 
been very much discussed. 

ANECA considers that the learning outcomes of the new programmes should be in line, at least, with: the naming of the proposed 
programme; the National Qualifications Framework and/or the Framework for Qualifications of the European Higher Education 
Area; the stakeholders’s opinion; and similar programmes of the international area that support this programme. With this 
criteria, several programmes of the Bachelor level did not receive a positive report from ANECA at the evaluation procedure, 
because they were more appropiate for a Master level. 

In this link, more related information can be consulted: http://www.enqa.eu/files/QA%20and%20LO/Laureano%20Gonzalez.pdf 


random choice of Bachelor’s and Master’s thesis against the information appearing in 
the proposal. 

2.4 Conclusions 

Learning outcomes have been one of the central topics of discussion in the ex-ante 
accreditation procedure that all Bachelor’s and Master’s degrees have been through 
during the last three years. If the focus here was on the intended learning outcomes, 
the follow-up and the ex-post accreditation procedures will focus on the achieved 
learning outcomes. 

This is not an easy task and there is a strong need for working out an agenda to help 
universities to adapt their teaching and learning methodologies to a learning outcomes 
approach (including their assessment). Several working groups have been established 
(with the help and support from quality assurance agencies, the Ministry of Education 
and the Bologna Experts Team in Spain) to monitor the role that learning outcomes are 
playing in our curricular reform (at the design and at the implementation levels) and, 
when implementing new degrees, to analyse how learning outcomes are used in order 
to identify common misunderstandings and to provide appropriate advice. 


Chapter 3: Determining whether 
intended learning outcomes meet 
subject-specific and academic and/or 
professional requirements 

Peter Cullen, Higher Education and Training Awards Council (HETAC), Ireland 


3.1. Meaning of Intended Learning Outcomes (ILOs) versus Expected 
Learning Outcomes (ELOs) versus Actual Learning Outcomes (ALOs) 

The following paragraphs outline understandings of the terminology used. 

Learning refers to the processes by which a sustainable change in someone’s 
knowledge, skill or competence occurs. 

Learning outcomes in this context means ‘statements of what a learner knows, 
understands and is able to do on completion of a learning process, which are defined in 
terms of knowledge, skills and competence’ (EQF^ definition). 

A programme of education and training refers to any process by which learners may 
acquire knowledge, skill or competence. 

Effective learning activities are purposefully directed towards attainment of a 
programme’s educational goals and build on (connect with) and modify prior learning. 

Intended learning outcomes represent a programme’s educational goals. They 
describe the learning outcomes that the teacher intends that learners will attain as 
a result of teaching and learning activities. It is reasonable to insist that intended 
learning outcomes must always include specification of the minimum intended 
learning outcomes. 

Actual learning outcomes represent the knowledge skill and competence at the end of 
a programme. They are actually achieved by a learner and should, if the programme is 
effective, include at least the minimum intended learning outcomes; they will typically 
include additional outcomes. 

Expected learning outcomes represent in a generalised way the level of learning 
expected before a particular type of qualification may be made - they are the concern 
of qualifications frameworks (and subject, discipline and professional standards). 
Expected learning outcomes may be expressed to define the minimum, typical or range 
of standards for the level - whichever it is, it should be made explicit. 

Learning outcomes may describe the change in knowledge, skill or competence 
or the cumulative result (end result) of all learning falling within the scope of the 
qualification and programme concerned, including prior learning at the time of entry 
to the programme. Expected and Intended learning outcomes expressed for major 
higher education awards (e.g. awards signifying completion of a Bologna Cycle) are 
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generally cumulative in this sense. Expected learning outcomes associated with 
qualifications frameworks are also generally cumulative. 

Ideally, intended learning outcomes should be assessable either directly or by 
inference. 

3.2. Mapping of Intended Learning Outcomes to European or National 
Qualifications Frameworks and any (other) legally binding standards 

It is common practice to map ILOs to qualifications frameworks and/or qualifications 
standards to determine whether they meet the legally binding subject-specific and 
academic and/or professional requirements expressed by these benchmarks. 

Before commenting on what is meant by mapping, it is important to reflect on how 
frameworks and other standards are expressed and on their varied purposes. 

Qualifications frameworks typically consist of a system of reference levels and may 
include qualification types. The reference levels of qualifications frameworks are 
normally expressed in terms of generalised expected learning outcomes statements 
(level indicators). Levels may be defined as ranges or as thresholds. Where a level is 
defined as a range, the indicators may, for example, signify level completion with pairs 
of such indicators bounding the range. Alternatively, indicators may express typical 
outcomes for the level. It is important to be clear about such matters as they can vary 
between different frameworks. 

Typically, the indicators’ elements are organised under common strands such as 
knowledge, skill or competence in the case of the European Qualifications Erameworks 
(EQE). Qualification types are templates for qualifications that are associated with a 
level. 

There are different kinds of qualifications frameworks, including international meta- 
frameworks, national qualifications frameworks and discipline-specific qualifications 
frameworks. 

Meta-frameworks are designed for aligning national qualifications frameworks but are 
frequently used to for aligning specific qualifications. They express statements about 
generalised knowledge, skill and competence. The EQE and Qualifications Eramework 
for the European Higher Education Area (QEEHEA) are examples. 

National qualifications frameworks are designed to set overall standards for 
qualifications, facilitate recognition of qualifications, and facilitate access to, 
and transfer and progression between programmes. There may be a single multi- 
purpose national qualifications framework in a country or there may be multiple 
national qualifications frameworks where each might, for example, be optimised for 
qualifications in a particular domain (e.g. vocational, higher education, etc.). European 
policy is that national qualifications frameworks be aligned to a meta-framework and 
hence to each other. 

Discipline specific qualifications frameworks may be national or international they 
are optimised for use within a discipline-area. 

Discipline-specific qualifications (or awards) standards may be established in 
association with a qualifications framework - these set out common features of 


specified qualification types in the relevant discipline. The QAA^ subject benchmarks 
are an example. 

The higher education qualifications required for certain professions are regulated. 
This may be at national level (by or through statute) or at international level (European 
Directives). It should be noted that not all legally binding standards are expressed 
as expected learning outcomes. Article 46 of Directive 2005/36/EC of the European 
Parliament and of the Council on the Recognition of Professional Qualifications (which 
deals with Architecture) is an example where learning outcomes are easily identifiable. 
However, there are many that focus on (or include) ‘inputs’ which have to do with how 
(and for how long) people are educated, trained and assessed. 

Mapping involves establishing a (level equivalence) link between the intended 
learning outcomes (ILOs) and the expected learning outcomes (ELOs) of a standard 
(whatever the type). 

Mapping is typically done in the context of developing, validating, accrediting, 
reviewing, approving, a programme or recognising a qualification. These processes can 
be found in both internal and external quality assurance procedures. 

Mapping is also applied in both institutional and system level research. 

3.3. Need to acknowledge the implications of uncertainty of such mappings— 
different experts may have different views 

Both intended learning outcomes (ILOs) and expected learning outcomes (ELOs) as 
statements about knowledge, skill and competence admit interpretation variance. 
Different disciplines may have subtly different ways of expressing ILOs and presumably 
interpreting ELOs. 

It has been argued in the literature that the concept of a ‘community of practice’ 
and the associated literature and ideas may be useful for helping to understand 
how practitioners in different disciplines interact with generalised standards such 
as qualifications frameworks and help define and express ILOs. It may also help 
understand how broader consensus about how to consistently interpret multi-purpose 
qualifications frameworks develops. 

Normally, prior to (or during) mapping the ILOs are expressed in (or transformed to) 
the same format as the relevant ELOs (for example, in knowledge, skill and competence 
strands). 

Mapping of ILOs, for example, to a level in a qualifications framework requires 
deep understanding of both the ILOs and the framework’s level indicators (ELOs). 
Mapping involves making judgements about level based on the ILOs and ELOs and 
one’s understanding of each informed by other evidence. It needs to be appreciated that 
judgements are informed by the indicators as well as a person’s experience of concrete 
examples (e.g. various qualifications with which she or he is familiar and which are 
linked with the framework and/or aligned frameworks). The communities of practice 
concept mentioned earlier may be helpful in understanding the issues. 

It is self-evident that judgements of even well-informed individuals may differ. 
Collective judgement perhaps informed by accumulation of individual judgements 
is warranted. This can be accomplished by survey, broad consultation or committee 
processes (involving panels of experts) or combinations of these. 

Regular reflection and evaluation of experience of implementing learning outcomes 
is also necessary (for example through national or international research projects). 
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3.4. Mapping (benchmarking) of ILOs against other internationally well- 
accepted standards for that subject or profession 

Comparing ILOs against other internationally well-accepted standards for the subject 
or profession is always warranted. This may include benchmarking a particular 
programme’s ILOs against those of internationally respected programmes at the 
same nominal level, including those in other countries. This is normally done when 
developing a new programme. 

Widespread use of qualifications frameworks and meta-frameworks can facilitate 
such comparisons by requiring programmes to express their ILOs in formats which 
facilitate international comparisons. 

It is very important for this benchmarking not to result in the cloning of 
programmes and qualifications. One needs to keep institutional autonomy and the need 
for diversity in mind. 

3.5. Mapping ILOs in emerging areas 

Standards development tends to lag the development of new disciplines. New 
disciplines such as Nano-science and Bio-informatics have grown from inter- 
disciplinary work. In new areas, there may be no inter-disciplinary standards and little 
consensus about specific ELOs. In such cases, the generic standards of qualifications 
framework provides some support as do discipline-specific standards of parent 
disciplines. Where the latter don’t exist, consistent mapping can be a significant 
challenge. 

In designing ILOs for new areas it is important to ensure that they are coherent and 
not mere aggregations of disconnected knowledge, skills and competences — mapping 
against generic qualifications frameworks can help bring such issues into focus. 

3.6. Panel functionality: needs to be able to work as team 

It has already been mentioned that mapping ILOs requires collective judgement. 
Typically, this will involve a panel of experts. The following principles are offered. 

The panel should include: 

• international expertise in the discipline concerned 

• pedagogical expertise 

• at least four members 

All who participate in such panels require training in how to consistently scrutinise 
ILOs and map them to the relevant standards. 

Training procedures may also be useful for the purpose of assessment and selection 
of experts. Performance of experts in panels should be monitored and this may also be 
used for future selection. 

When forming a panel, it is important that due regard is paid to the importance of 
the panel being able to function as a team. 

3.7. Scrutinising Intended Learning Outcomes (i.e. statements about 
knowledge, skill and competence) in the context of mapping 

When a panel (particularly, if it is external) is verifying a mapping of ILOs to a 
standard, the following points should be taken into account: 

• ILOs as statements may be incomplete 


• One needs to be clear about what the minimum ILOs are 

• Ways of expressing ILOs can vary between different disciplines 

• The existence of mature communities of practice is important to ensure ILOs will 
be interpreted and implemented consistently 

• Experience shows that academics are often weak on expressing skill and 
competence ILOs relative to knowledge ILOs 

• When scrutinising ILOs, one should not restrict ones attention to the ILO 
statements but should also look at assessment (formative and summative), 
teaching, etc., which can aid in their interpretation and perhaps reveal tacit 
outcomes or the reverse 

• Professional associations tend to be relatively strong on the expression of skills 
and competences 

Examination of the associated programme’s inputs, environment, intermediate 
outcomes, assessment tasks, etc., can all help better understand the intended learning 
outcomes and the real standard of knowledge, skill and competence required behind 
the ILOs to be mapped. 

Here it is assumed that knowledge, skill and competence in principle can be 
described. The learning outcomes perspective is particularly well suited to thinking 
about teaching, learning and quality assurance and is strongly to be encouraged. All of 
this should not blind one to the profound dangers of focusing exclusively on statements 
about knowledge, skill and competence or relying solely on comparisons between 
such abstractions. Any statement about knowledge, skill and competence must be 
backed up by evidence and this does not mean the existence of good quality assurance 
procedures — it means direct evidence. In some cases, the interpretation of an intended 
learning outcome can be clarified if the conditions which are expected to result in that 
learning, or the assessment task which must be successfully accomplished, are also 
described. 

Higher education quality assurance has never eschewed direct examination of such 
matters. Accreditation or validation (i.e. external programme approval) is an example 
of a quality assurance procedure which is concerned with inputs, environment and 
outputs. 

Einally a caveat: Learning outcomes do not represent that last word on curriculum or 
qualification design and indeed there is ongoing debate about the limitations of LOs as 
a conceptual tool. 


Chapter 4: Which requirements 
should the formulation of learning 
outcomes meet? 


Norma Ryan, University College Cork - National University of Ireland Cork, Ireland 

4.1. Introduction 

The introduction of a learning outcomes based approach to teaching in higher 
education as a formal replacement to the teaching-centred approach was an intrinsic 
part of the implementation of the Bologna Process. This has essentially meant a shift 
in focus from the teacher to the student with an emphasis on what a student should 
know, understand or be able to demonstrate after completion of a process of learning. 
Although in many instances and for many individual teachers, this has not meant a 
total shift in approach or in pedagogies used, it is a shift that has required effort on the 
part of institutions and many teachers. Indeed, the students also needed and continue 
to need to adapt to the central shift and to take ownership for the learning journey on 
which they are embarked. 

At this point in time, there is a very good, general agreement as to what the 
definition of a learning outcome is: 


Learning outcomes are statements of what a student should know, understand 
and! or be able to demonstrate after completion of a process of learning 


The learning activity could be, for example, a lecture, a module or an entire 
programme. For compliance with the Bologna Process, including the alignment of 
academic programmes with national qualifications frameworks, learning outcomes 
must be written for each module and for each programme. Learning outcomes must 
be simply and clearly described and must be capable of being validly assessed. From 
this definition of learning outcomes, the observer can see an emphasis on the learner 
and on the learner’s ability to ‘do something. This is a central shift in focus away from 
the teacher and his/her objectives and towards the student and his/her achievements 
and the skills/competences acquired following engagement in a piece of learning. The 
use of active verbs and terms such as define, list, name, recall, analyse, calculate, design, 
etc. to describe the learning outcomes, facilitate the explanation of the achievement 
targets to be met by the student. It is important to note that learning outcomes are 
not designed to replace the traditional way of describing teaching and learning but to 
supplement it. 

The subject of a definition of competence or competency is another question and will 
not be discussed in this brief paper. 


4.2. Experiences in implementing a Learning Outcomes Approach - 
University College Cork: A case study 

One of the key issues arising from this shift in emphasis in approach and attitude 
in higher education is the need to quality assure the process - the pedagogies, the 
assessment methodologies and the student learning undertaken. This involves, inter 
alia, a re-consideration of all curricula and the programmes associated with that 
curriculum. 

The paper presented at the ENQA Workshop - Quality Assurance and Learning 
Outcomes, Vienna, Austria, 9-10 September 2010, focused on the experience of 
University College Cork (UCC) in introducing the learning outcomes approach to 
teaching across the entire university in a systematic way, and the steps taken to ensure 
that the introduction was accompanied by appropriate rigour and standards. 

Initially, the university participated in an EUA Quality Culture III Project on 
Implementing a Learning Outcomes based approach to teaching in 2003-2005^. A 
network of six EU universities was involved and the participation of UCC in the project 
was led by Prof. Aine Hyland, Vice-president and Professor of Education and Dr Norma 
Ryan, Quality Promotion Unit, UCC. A project of eighteen month duration, the final 
report was published in 2005. The project focussed on Learning Outcomes rather than 
Competences, and prepared the way in UCC for the introduction of the approach. 

Eollowing on the completion of the EUA project activities, an international 
Conference on learning outcomes was organised in 2005 in UCC^°. In addition to a 
panel of international experts, a significant part of the conference was the presentation 
of exemplar material from academic staff of UCC and other institutions on their 
experiences and research into the experience and effects of trialling the learning 
outcomes approach to their teaching. Many of the contributors from UCC were engaged 
in the process of studying themselves for the UCC accredited Postgraduate Certificate 
or Diploma in Teaching and Learning in Higher Education. 

Eollowing on from that event, a plan was prepared for the stepwise implementation 
of the learning outcomes based approach across all programmes - both undergraduate 
and postgraduate - offered by the university and based on these experiences. This plan 
was considered by faculties and ultimately approved by Academic Council following the 
collegial discussions and consultations. A key resource to this activity was the expertise 
developed by the contributors to the conference in conducting the research prior to the 
presentation of their work. Seminars were held in individual departments and faculties 
and mentoring of staff also took place for those interested. 

One of the key elements to the success of the implementation programme in UCC 
was the publication of a handbook Writing and using learning Outcomes - a Practical 
Guide^^ by an expert in the field and a colleague in UCC, Dr. Declan Kennedy. The 
guidelines detailed in the handbook were adopted by the Academic Council of the 
university and served as the rule book for staff. The handbook included a simple and 
clear set of guides for the terminology to be used, and the recommended level of detail 
to be included in formal university publications. 

Accompanying this guide the University’s Teaching and Learning Support Centre, 
lonad Bairre, provided on-going support via lunchtime seminars and workshops on the 
aspects of the introduction of the approach. These sessions were open to all interested 
academic staff to attend and discussion was welcomed. Topics included assessment 


9 http://www.eua.be/eua/jsp/en/upload/Quality_Culture_2002_2003.1150459570109.pdf 

10 http://www.ucc.ie/en/qpu/loc/. All papers and presentations may be accessed here. 

11 Published by UCC in 2007, © Quality Promotion Unit. ISBN 978-0-9552229-6-2. 


methodologies, pedagogies, assuring that achievement of the learning outcomes were 
all covered. 

The university prospectus and calendar are published on the university website. 
These documents contain full descriptions for all modules and programmes according 
to a set of guides laid down by the university^^. All programmes offered by the 
university were required to be described in a specific manner and according to a 
particular format. This process was overseen by a subcommittee of Academic Council, 
charged with ensuring the rigour and correctness of the procedures. The format 
adopted by the university ensured the inclusion of learning outcomes and assessments 
for all modules offered, accompanied by a clear description of the module content and 
other relevant information. 

The figure below indicates the cycle of reflection encouraged in all staff to ensure an 
on-going reflection and improvement in the module descriptors, including the learning 
outcomes and also the pedagogies and assessment methodologies. 



1. IDENTIFY AIMS AND OBJECTIVES OF MODULE 



2. WRITE LEARNING OUTCOMES USING STANDARD GUIDELINES 



3. DEVELOP A TEACHING AND LEARNING STRATEGY TO ENABLE STUDENTS TO ACHIEVE 

LEARNING OUTCOMES 



4. DESIGN ASSESSMENT TO CHECK IF LEARNING OUTCOMES HAVE BEEN ACHIEVED 



5. IF NECESSARY, MODIFY MODULE CONTENT AND ASSESSMENT IN LIGHT OF FEEDBACK 


Staff were, and continue to be, actively encouraged to reflect on their teaching 
practices and to improve as appropriate. All aspects of curriculum design, delivery, 
assessment and standards need to be reflected upon in the adoption of a learning 
outcomes approach. 

Students were also involved in the research and activities leading up to the adoption 
of this approach in UCC. It was actively welcomed by the students who appreciated the 
clarity and transparency this brought to the objectives of their studies and curriculum. 
Employers and other stakeholders also welcomed the efforts by the university to clearly 
and transparently detail all aspects of programme delivery - in particular at the 
module level. 


12 http://www.ucc.ie/en/CurrentStudents/ 







A key objective for the university was alignment of the programmes with the 
National Qualifications Framework^^. This objective was seen as very important by the 
university and is a key element of institutional reviews in Ireland. The setting down 
of the learning outcomes was essential to this process of changing and improving the 
traditional ways of describing qualifications and qualification structures. 

Key considerations for quality assurance at the module and programme levels 
include: 

• Are the learning outcomes written according to the guidelines in the literature? 

• Are the learning outcomes written according to the guidelines laid down by the 
institution? 

The institution has a responsibility to oversee the process of introduction of a learning 
outcomes approach to teaching and to be able to assure its stakeholders that: 

1. Learning outcomes are described for modules and programmes 

2. Learning outcomes reach a certain standard 

3. Students and staff are informed of the learning outcomes to be achieved in an 
accessible manner 

4. Staff are supported and guided in the writing of learning outcomes 

5. The institution is satisfied that the approach of staff is not tokenistic 

6. The quality of the learning outcomes is evaluated 

7. The learning outcomes for modules and programmes are consistent with the 
National Qualifications Framework. 

4.3. Conclusion 

This paper presents a summary of the approach taken in UCC to the introduction of 
a learning outcomes approach to teaching and learning. The approach was based on 
both a top-down and bottom-up approach with senior management of the university 
supporting and leading the discussions whilst allowing a collegial approach to the 
introduction, engaging the wider academic community in the decisions to be taken. 
The introduction of this approach was very successful in UCC and has led to an 
embedded system endorsed by both staff and students within the university^"^. 
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ENQA Workshop: ''Quality Assurance and Learning Outcomes" 

9-10 September 2010 

Hosted by the Austrian FH Council 


VENUE 

Hilton Vienna Danube 
Handelskai 269 
1020 Vienna, Austria 


Objectives / Questions 

• What are the opportunities and the challenges of learning outcome orientation in 
the higher education sector? 

• What do the various stakeholders expect from quality assurance agencies in 
connection with learning outcome orientation? 

• In which way can or should learning outcomes be considered within the scope of 
external quality assurance? What role should “learning outcomes” play in external 
quality assurance? 


PROGRAMME 


Thursday 9 September 

12.00 Registration and light welcome lunch buffet 

Chair: Kurt Sohm, Managing Director of FHR, ENQA Board member 

13.15 Background/motives, opportunities and challenges of the shift towards 
“learning outcomes” and the possible impact on quality assurance 
agencies. 

Don Westerheijden, CHEFS 

13.45 The Students’ Perspective: Pros and cons of “learning outcomes”. 

Expectations towards quality assurance agencies. 

Robert Santa, ESU 

14.15 Perspective of the higher education institutions: Challenges of 
“learning outcome orientation” for quality assurance of higher education 
institutions. Interaction between internal and external quality assurance. 
Volker Gehmlich, EUA 

14.45 Perspective of the business world/employers: What is expected from the 
higher education institutions and quality assurance agencies? 

Oliver Maassen, Business Europe 

15.30 Coffee break 

Chair: Rafael Llavori, ANECA, ENQA Board member 

16.00 Moderated panel discussion: Is there a conflict of interest between labour 
market orientation and higher education values when using a learning 
outcome approach? 

Participants: 

Volker Gehmlich, EUA; 

Bardhyl Musai, Albanian Accreditation Council of Higher Education; 

Robert Santa, ESU; 

Oliver Maassen, Business Europe; 

Achim Hopbach, President of ENQA 

18.00 End of the first day 


Friday 10 September 

Chair: Laura Beccari, OAQ 

09.00 Three examples of considering “learning outcomes” in external 
quality assurance 

ANECA - Laureano Gonzalez 
NVAO - Mark Frederiks 

Stockholm University and Royal Technical College KTH - Lena Adamson 

10.30 Coffee break 

11.00 An example of common principles in the consideration of “learning 
outcomes”? - ECA principles on learning outcomes 

Herwig Patscheider, FHR 

12.00 Lunch break 

13.30 Parallel working groups - 

How are learning outcomes to be considered in external quality assurance 
processes? 

Working group A. (Room: TULLN) - How can it be determined whether 
the intended learning outcomes meet subject-specific and academic and/or 
professional requirements (expertise of the review team, international 
comparisons (benchmarks), technical qualification frameworks, etc.)? 

Peter Cullen, HETAC 

Working group B. (Room: LINZ/MELK) - How can it be determined 
whether the intended learning outcomes are met (assessment of final papers, 
interviews of graduates, checking the consistency of learning outcomes - 
contents - methods of teaching and learning - examination methods, etc.)? 
Obe de Vries, Dutch Education Inspectorate 

Working group C. (Room: HAINBURG) - Which requirements should the 
formulation of learning outcomes meet? 

Norma Ryan, University College Cork 

15.00 Break 

Chair: Kurt Sohm, Managing Director of FHR, ENQA Board member 

15.15 Presentation and discussion of the results of the working groups - 

further steps 

Anne Rdsdnen, University of Jyvaskyla 

16:00 End of the workshop 

Achim Hopbach, President of ENQA 




This report is based on the ENQA workshop on the theme “Quality Assurance 
and Learning Outcomes” that was held in September 2010 in Vienna, Austria. 

The workshop addressed the question of what stakeholders expect from quality 
assurance agencies in connection with learning outcome orientation. The 
opportunities and challenges of the learning outcome orientation in the higher 
education sector were discussed from different perspectives. The workshop tried to 
define the role that learning outcomes should play in external quality assurance and 
how they can or should be considered within the scope of external quality assurance. J 
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